The Moses–Littenberg meta-analytical method generates systematic differences in test accuracy compared to hierarchical meta-analytical models

AbstractObjectiveTo compare meta-analyses of diagnostic test accuracy using the Moses–Littenberg summary receiver operating characteristic (SROC) approach with those of the hierarchical SROC (HSROC) model.Study Design and SettingTwenty-six data sets from existing test accuracy systematic reviews were reanalyzed with the Moses–Littenberg model, using equal weighting (“E-ML”) and weighting by the inverse variance of the log DOR (“W-ML”), and with the HSROC model. The diagnostic odds ratios (DORs) were estimated and covariates added to both models to estimate relative DORs (RDORs) between subgroups. Models were compared by calculating the ratio of DORs, the ratio of RDORs, and P-values for detecting asymmetry and effects of covariates on DOR.ResultsCompared to the HSROC model, the Moses–Littenberg model DOR estimates were a median of 22% (“E-ML”) and 47% (“W-ML”) lower at Q*, and 7% and 42% lower at the central point in the data. Instances of the ML models giving estimates higher than the HSROC model also occurred. Investigations of heterogeneity also differed; the Moses–Littenberg models on average estimating smaller differences in RDOR.ConclusionsMoses–Littenberg meta-analyses can generate lower estimates of test accuracy, and smaller differences in accuracy, compared to mathematically superior hierarchical models. This has implications for the usefulness of meta-analyses using this approach. We recommend meta-analysis of diagnostic test accuracy studies to be conducted using available hierarchical model–based approaches

Test-treatment RCTs are susceptible to bias:a review of the methodological quality of randomized trials that evaluate diagnostic tests

Abstract Background There is a growing recognition for the need to expand our evidence base for the clinical effectiveness of diagnostic tests. Many international bodies are calling for diagnostic randomized controlled trials to provide the most rigorous evidence of impact to patient health. Although these so-called test-treatment RCTs are very challenging to undertake due to their methodological complexity, they have not been subjected to a systematic appraisal of their methodological quality. The extent to which these trials may be producing biased results therefore remains unknown. We set out to address this issue by conducting a methodological review of published test-treatment trials to determine how often they implement adequate methods to limit bias and safeguard the validity of results. Methods We ascertained all test-treatment RCTs published 2004–2007, indexed in CENTRAL, including RCTs which randomized patients to diagnostic tests and measured patient outcomes after treatment. Tests used for screening, monitoring or prognosis were excluded. We assessed adequacy of sequence generation, allocation concealment and intention-to-treat, appropriateness of primary analyses, blinding and reporting of power calculations, and extracted study characteristics including the primary outcome. Results One hundred three trials compared 105 control with 119 experimental interventions, and reported 150 primary outcomes. Randomization and allocation concealment were adequate in 57 and 37% of trials. Blinding was uncommon (patients 5%, clinicians 4%, outcome assessors 21%), as was an adequate intention-to-treat analysis (29%). Overall 101 of 103 trials (98%) were at risk of bias, as judged using standard Cochrane criteria. Conclusion Test-treatment trials are particularly susceptible to attrition and inadequate primary analyses, lack of blinding and under-powering. These weaknesses pose much greater methodological and practical challenges to conducting reliable RCT evaluations of test-treatment strategies than standard treatment interventions. We suggest a cautious approach that first examines whether a test-treatment intervention can accommodate the methodological safeguards necessary to minimize bias, and highlight that test-treatment RCTs require different methods to ensure reliability than standard treatment trials. Please see the companion paper to this article: http://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-016-0286-0

Exfoliative cytology for diagnosing basal cell carcinoma and other skin cancers in adults

Background: Early accurate detection of all skin cancer types is essential to guide appropriate management and to reduce morbidity and improve survival. Basal cell carcinoma (BCC) is usually localised to the skin with potential to infiltrate and damage surrounding tissue, while cutaneous squamous cell carcinoma (cSCC) and melanoma have a much higher potential to metastasise and ultimately lead to death. Exfoliative cytology is a non–invasive test that uses the Tzanck smear technique to identify disease by examining the structure of cells obtained from scraped samples. This simple procedure is a less invasive diagnostic test than a skin biopsy, and for BCC has the potential to provide an immediate diagnosis that avoids an additional visit to clinic to receive skin biopsy results. This may benefit patients scheduled for either Mohs micrographic surgery or non–surgical treatments such as radiotherapy. A cytology scrape can never give the same information as a skin biopsy, however, so it is important to know more about which skin cancer situations it may be helpful.Objectives: The primary objective was to determine the diagnostic accuracy of exfoliative cytology for the detection of basal cell carcinoma (BCC) in adults. Secondary objectives were to determine diagnostic accuracy for the detection of i) cutaneous squamous cell carcinoma, ii) invasive melanoma and atypical intraepidermal melanocytic variants, and iii) any skin cancer, including keratinocyte skin cancer, invasive melanoma and atypical intraepidermal melanocytic variants, or any other skin cancer.Search methods: We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials; MEDLINE; EMBASE; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We also studied the reference lists of published systematic review articles.Selection criteria: Studies evaluating exfoliative cytology in adults with lesions suspicious for BCC, cSCC or melanoma, compared with a reference standard of histological confirmation.Data collection and analysis: Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS-2). Where possible we estimated summary sensitivities and specificities using the bivariate hierarchical model.Main results: This review reports on nine studies with a total of 1655 lesions including 1120 BCCs (14 datasets), 401 lesions with 44 cSCCs (two datasets), and 200 lesions with 10 melanomas (one dataset). Three of these datasets (one each for BCC, melanoma, and any malignant condition) were derived from one study which also performed a direct comparison with dermoscopy. Studies were of moderate to poor quality providing inadequate descriptions of participant selection, thresholds used to make cytological and histological diagnoses, and blinding. Reporting of patients’ prior referral pathways was particularly poor, as were descriptions of the cytodiagnostic criteria used to make diagnoses. No studies evaluated the use of exfoliative cytology as a primary diagnostic test for detecting BCC or other skin cancers in lesions suspicious for skin cancer. Pooled data from seven studies using standard cytomorphological criteria (but various stain methods) to detect BCC in patients with a high clinical suspicion of BCC estimated the sensitivity and specificity of exfoliative cytology as 97.5% (95% CI: 94.5 to 98.9%) and 90.1% (95% CI: 81.1 to 95.1%) respectively. When applied to a hypothetical population of 1000 clinically suspected BCC lesions with a median observed BCC prevalence of 86%, exfoliative cytology would miss 21 BCCs and would lead to 14 false positive diagnoses of BCC. No false positive cases were histologically confirmed to be melanoma. Insufficient data are available to make summary statements regarding the accuracy of exfoliative cytology to detect melanoma or cSCC, or its accuracy compared to dermoscopy.Authors' conclusions: The utility of exfoliative cytology for the primary diagnosis of skin cancer is unknown, as all included studies focused on the use of this technique for confirming strongly suspected clinical diagnoses. For the confirmation of BCC in lesions with a high clinical suspicion, there is evidence of high sensitivity and specificity for exfoliative cytology. Since decisions to treat low riskBCCs are unlikely in practice to require diagnostic confirmation given that clinical suspicion is already high, exfoliative cytology might be most useful for cases of BCC where the treatments being contemplated require a tissue diagnosis (e.g. radiotherapy). The small number of included studies, poor reporting and varying methodological quality means that no strong conclusions can currently be drawn to guide clinical practice. Despite insufficient data on the use of cytology for cSCC or melanoma, it is unlikely that cytology would be useful in these scenarios since preservation of the architecture of the whole lesion that would be available from a biopsy provides crucial diagnostic information. Given the paucity of good quality data, appropriately designed prospective comparative studies may be required to evaluate both the diagnostic value of exfoliative cytology by comparison to dermoscopy, and its confirmatory value in adequately reported populations with a high probability of BCC scheduled for further treatment requiring a tissue diagnosis

Management of endocrine disease:Imaging for the diagnosis of malignancy in incidentally discovered adrenal masses: a systematic review and meta-analysis

Objective: Adrenal masses are incidentally discovered in 5% of CT scans. In 2013/2014, 81 million CT examinations were undertaken in the USA and 5 million in the UK. However, uncertainty remains around the optimal imaging approach for diagnosing malignancy. We aimed to review the evidence on the accuracy of imaging tests for differentiating malignant from benign adrenal masses. Design: A systematic review and meta-analysis was conducted. Methods: We searched MEDLINE, EMBASE, Cochrane CENTRAL Register of Controlled Trials, Science Citation Index, Conference Proceedings Citation Index, and ZETOC (January 1990 to August 2015). We included studies evaluating the accuracy of CT, MRI, or F-18-fluoro-deoxyglucose (FDG)-PET compared with an adequate histological or imaging-based follow-up reference standard. Results: We identified 37 studies suitable for inclusion, after screening 5469 references and 525 full-text articles. Studies evaluated the accuracy of CT (n = 16), MRI (n = 15), and FDG-PET (n = 9) and were generally small and at high or unclear risk of bias. Only 19 studies were eligible for meta-analysis. Limited data suggest that CT density >10 HU has high sensitivity for detection of adrenal malignancy in participants with no prior indication for adrenal imaging, that is, masses with <= 10 HU are unlikely to be malignant. All other estimates of test performance are based on too small numbers. Conclusions: Despite their widespread use in routine assessment, there is insufficient evidence for the diagnostic value of individual imaging tests in distinguishing benign from malignant adrenal masses. Future research is urgently needed and should include prospective test validation studies for imaging and novel diagnostic approaches alongside detailed health economics analysis

Accuracy of routine laboratory tests to predict mortality and deterioration to severe or critical COVID-19 in people with SARS-CoV-2

Objectives This is a protocol for a Cochrane Review (diagnostic). The objectives are as follows:  To assess the accuracy of routine blood-based laboratory tests to predict mortality and deterioration to severe or critical (from mild or moderate) COVID-19 in people with SARS-CoV-2 infection. Secondary objectives Where data are available, we will investigate whether prognostic accuracy varies according to a specific measurement or test, reference standard, timing of outcome verification, sample type, study design, and setting, including prevalence of the target condition (either by stratified analysis or meta-regression)

The use of teledermatology for the diagnosis of skin cancer in adults

Background: Early accurate detection of all skin cancer types is essential to guide appropriate management and to improve morbidity and survival. Melanoma and squamous cell carcinoma (SCC) are high risk skin cancers which have the potential to metastasise and ultimately lead to death, whereas basal cell carcinoma (BCC) is usually localised with potential to infiltrate and damage surrounding tissue. Anxiety around missing early curable cases needs to be balanced against inappropriate referral and unnecessary excision of benign lesions. Teledermatology provides a way for generalist clinicians to access the opinion of a specialist dermatologist for skin lesions that they consider to be suspicious without referring the patients concerned through the normal referral pathway. Teledermatology consultations can be ‘store-and-forward’ with electronic digital images of a lesion sent to a dermatologist for review at a later time, or can be live and interactive consultations using video conferencing to connect the patient, referrer and dermatologist in real time. Objectives: To determine the diagnostic accuracy of teledermatology for the detection of any skin cancer (melanoma, BCC or cSCC) in adults, and to compare its accuracy with that of in-person diagnosis. Search methods: We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials; MEDLINE; EMBASE; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We studied reference lists and published systematic review articles. Selection criteria: Studies evaluating skin cancer diagnosis for teledermatology alone, or in comparison with face-to-face diagnosis by a specialist clinician, compared with a reference standard of histological confirmation or clinical follow-up and expert opinion. Studies evaluating the referral accuracy of teledermatology compared with a reference standard of face-to-face diagnosis by a specialist clinician were also included. Data collection and analysis: Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS-2). We contacted authors of included studies where information related to the target condition of any skin cancer was missing. Data permitting, we estimated summary sensitivities and specificities using the bivariate hierarchical model. Due to scarcity of data, no covariate investigations were undertaken for this review. For illustrative purposes, estimates of sensitivity and specificity were plotted on coupled forest plots for diagnostic threshold and target condition under consideration. Main results: Twenty-two studies were included reporting diagnostic accuracy data for 4057 lesions and 879 malignant cases (16 studies) and referral accuracy data for reported data for 1449 lesions and 270 ‘positive’ cases as determined by the reference standard face-to-face decision (six studies). Methodological quality was variable with poor reporting hindering assessment. The overall risk of bias was rated as high or unclear for participant selection, reference standard and participant flow and timing in at least half of all studies; the majority were considered at low risk of bias for the index test. The applicability of study findings were of high or unclear concern for the majority of studies in all domains assessed due to the recruitment of study participants from secondary care settings or specialist clinics rather than from the primary or community-based settings in which teledermatology is more likely to be used and due to the acquisition of lesion images by dermatologists or in specialist imaging units rather than by primary care clinicians. Seven studies provided data for the primary target condition of any skin cancer (1588 lesions and 638 malignancies). For the correct diagnosis of lesions as malignant using photographic images, summary sensitivity was 94.9% (95% CI 90.1 to 97.4%) and summary specificity 84.3% (95% CI 48.5 to 96.8%) (from four studies). Individual study estimates using dermoscopic images or a combination of photographic and dermoscopic images generally suggested similarly high sensitivities with highly variable specificities. Limited comparative data suggested similar diagnostic accuracy between teledermatology assessment and in-person diagnosis by a dermatologist; however, data were too scarce to draw firm conclusions. For the detection of invasive melanoma or atypical intraepidermal melanocytic variants both sensitivities and specificities were more variable. Sensitivities ranged from 59% (95% CI 42% to 74%) to 100% (95% CI 54% to 100%) and specificities from 30% (95% CI 22% to 40%) to 100% (95% CI 93% to 100%), with reported diagnostic thresholds including the correct diagnosis of melanoma, classification of lesions as ‘atypical’ or ‘typical as well as the decision to refer or to excise a lesion. Referral accuracy data comparing teledermatology against a face-to-face reference standard suggested good agreement for lesions considered to require some positive action by face to face assessment (sensitivities of over 90%). For lesions considered of less concern when assessed face-to-face (e.g. for those not recommended for excision or referral), agreement was more variable with teledermatology specificities ranging from 57% (95% CI 39 to 73%) to 100% (95% CI 86% to 100%), suggesting that remote assessment is more likely recommend excision, referral or follow-up compared to in-person decisions. Authors' conclusions: Studies were generally small and heterogeneous and methodological quality was difficult to judge due to poor reporting. Bearing in mind concerns regarding the applicability of study participants and of lesion image acquisition in specialist settings, our results suggest that teledermatology can correctly identify the majority of malignant lesions. Using a more widely defined threshold to identify ‘possibly’ malignant cases or lesions that should be considered for excision is likely to appropriately triage those lesions requiring face-to-face assessment by a specialist. Despite the increasing use of teledermatology on an international level, the evidence base to support its ability to accurately diagnose lesions and to triage lesions from primary to secondary care is lacking and further prospective and pragmatic evaluation is needed

Smartphone applications for triaging adults with skin lesions that are suspicious for melanoma

Background: Melanoma accounts for a small proportion of all skin cancer cases but is responsible for the majority of skin cancer-related deaths. Early detection and treatment can improve survival. Smartphone applications are readily accessible and potentially offer an instant risk assessment of the likelihood of malignancy, so that the right people seek further medical attention from a clinician for more detailed assessment of the lesion. There is, however, a risk that melanomas will be missed and treatment delayed if the application reassures the user that their lesion is low risk. Objectives: To determine the diagnostic accuracy of smartphone applications to rule out cutaneous invasive melanoma and intraepidermal melanocytic variants in adults with concerns about suspicious skin lesions. Search methods: We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials; MEDLINE; Embase; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We studied reference lists and published systematic review articles. Selection criteria: Studies of any design evaluating smartphone applications intended for use by individuals in a community setting who have lesions that might be suspicious for melanoma or intraepidermal melanocytic variants compared with a reference standard of histological confirmation or clinical follow-up and expert opinion. Data collection and analysis: Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS-2). Due to scarcity of data and poor quality of studies, no meta-analysis was undertaken for this review. For illustrative purposes, estimates of sensitivity and specificity were plotted on coupled forest plots for each application under consideration. Main results: This review reports on two cohorts of lesions published in two studies. Both studies were at high risk of bias from selective participant recruitment, and high rates of non-evaluable images. Concerns about applicability of findings were high due to inclusion only of lesions already selected for excision in a dermatology clinic setting, and image acquisition by clinicians rather than by smartphone app users. Data for five mobile phone applications were reported for 332 suspicious skin lesions with 86 melanomas across the two studies. Across the four artificial intelligence-based applications which classified lesion images (photographs) as melanomas (one application) or as high risk or ‘problematic’ lesions (three applications) using a pre-programmed algorithm, sensitivities ranged from 7% (95% CI: 2%, 16%) to 73% (95% CI: 52%, 88%) and specificities from 37% (95% CI: 29% to 46%) to 94% (95% CI: 87%, 97%). The single application using store-and-forward review of lesion images by a dermatologist had a sensitivity of 98% (95% CI: 90%, 100%) and specificity 30% (95% CI: 22%, 40%). The number of test failures (lesion images analysed by the applications but classed as ‘not evaluable’ and excluded by the study authors) ranged from 3 to 31 (or 2% to 18% of lesions analysed). The store-and-forward application had one of the highest rates of test failure (15%). At least one melanoma was classed as ‘not evaluable’ in three of the four application evaluations. Authors' conclusions: Smartphone applications using artificial intelligence-based analysis have not yet demonstrated sufficient promise in terms of accuracy, and are associated with a high likelihood of missing melanomas. Applications based on store-and-forward images could have a potential role in the timely presentation of people with potentially malignant lesions by facilitating active self-management health practices and early engagement of those with suspicious skin lesions; however, they may incur a significant increase in resource and workload. Given the paucity of evidence and low methodological quality, no implications for practice can be drawn. Nevertheless, this is a rapidly advancing field and new and better applications with robust reporting of studies could change these conclusions substantially